Repository navigation
Conversation
The coverage job checks out the merge of the PR head into the base branch, but codecov-action uploads the report under the PR head SHA. When the base branch has commits the PR lacks, Codecov reads the merge tree's coverage at the head's line numbers and flags the wrong lines. Same change as apache/datafusion-sqlparser-rs#2832. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
alamb
marked this pull request as ready for review
November 7, 2026 21:18
Contributor
Author
|
I can't make sense of the codecov report. Closing for now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
The Codecov reports on our PRs are unreliable: they regularly mark comments and blank lines as "Missing" and show unexecuted code as covered (exmaples below)
The problem is that the coverage job compares the PR to the base branch (e.g. main), but Codecov attributes the report to the PR head commit
Examples of Codecov results that refer to lines not in the PR
For each PR below I took Codecov's per-line coverage for the head commit (via the Codecov API), the PR diff (merge-base..head) and the merge tree the job actually built (
git merge-treeof the head and themaintip at the time the run was created).#25455 (
datafusion/optimizer/src/push_down_filter.rs): the Codecov comment reports70.73% | 15 Missing and 9 partials.See the report here.
Among the lines it marks as Missing are L4593-L4594, L4613-L4615, which are a
///doc comment, a blank line and two//comments:Out of 243 PRs merged to
mainbetween 2026-09-25 and 2026-10-20 that touch Rust files, 26 had at least one changed file whose coverage was read at shifted line numbers.Why the line numbers are wrong, and which PRs are affected today
For a
pull_requestrun,actions/checkoutchecks out the merge of the PR head into the base branch, butcodecov-actionuploads the report under the PR head SHA. Whenever the base branch has commits the PR lacks, Codecov reads the merge tree's coverage at the head's line numbers.Today this affects
pull_requestruns in theapacherepo: PRs tobranch-*(backports) and non-fork PRs tomain. Fork PRs tomainrun in the fork onpushsince #25591, where the head is already checked out, so they are not affected. Before #25591 every PR was affected, which is where the examples above come from.What changes are included in this PR?
Update coverage job to check out
github.event.pull_request.head.shaonpull_requestruns, andgithub.shaotherwise (merge queue, pushes tobranch-*).The trade-off is the same as in the sqlparser PR: the
cargo test (amd64)job now tests the PR head rather than the merge with the base branch; I actually think this is more intuitive as it is the same commit that gets checked out locally.Note that the merge queue still tests the merged result before anything lands on
main.What is the testing strategy for this PR?
CI on this PR. The Codecov comment on this PR cannot show the difference since it changes no Rust code; the examples above were verified against Codecov's per-line data and the reconstructed merge trees.
Are there any user-facing changes?
No.